Add: report NEXT_LEVEL reservation stalls - #1613
Conversation
d688ef9 to
1a3a55c
Compare
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThe scheduler now uses wake generations and completion events for waiting. NEXT_LEVEL dispatch returns reservation details, tracks structural stalls, and emits timed diagnostics. Queue inspection and tests cover blocked groups, queued singles, duplicate completions, and wake-up behavior. ChangesScheduler wake and reservation stall handling
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Scheduler
participant ReadyQueues
participant CompletionFIFO
participant WorkerDiagnosticSink
Scheduler->>ReadyQueues: inspect target queue heads
ReadyQueues-->>Scheduler: reservation and queue state
Scheduler->>Scheduler: track wake generation and stall deadline
CompletionFIFO-->>Scheduler: terminal completion
Scheduler->>WorkerDiagnosticSink: report reservation stall
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
1a3a55c to
3b3ebb1
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (4)
tests/ut/cpp/hierarchical/test_scheduler.cpp (2)
1179-1186: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low valueAssert the worker pointers before dereferencing them.
manager.get_worker_by_idreturnsnullptrfor an unregistered id.Scheduler::dispatch_next_level_groupinsrc/common/hierarchical/scheduler.cppchecks for that case explicitly. Worker ids 0 and 1 are registered inSetUp(), so the current test is safe. AddASSERT_NE(..., nullptr)so a future registration change fails as an assertion instead of a segmentation fault inside a lock scope.🛡️ Proposed change
WorkerThread *manager_worker_a = manager.get_worker_by_id(WorkerType::NEXT_LEVEL, 0); WorkerThread *manager_worker_b = manager.get_worker_by_id(WorkerType::NEXT_LEVEL, 1); + ASSERT_NE(manager_worker_a, nullptr); + ASSERT_NE(manager_worker_b, nullptr); while ((!manager_worker_a->idle() || !manager_worker_b->idle()) &&🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/ut/cpp/hierarchical/test_scheduler.cpp` around lines 1179 - 1186, Add ASSERT_NE checks for manager_worker_a and manager_worker_b immediately after their get_worker_by_id calls and before the idle() polling loop, so null worker registrations fail the test safely before dereferencing either pointer.
786-796: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winGate the diagnostic field checks with an assertion.
If the report does not arrive within 200 ms, line 790 fails and the test continues. Lines 791-796 then compare default-initialized capture fields and emit five more failures. Use
ASSERT_EQfor the report-count check so the test stops at the root cause.The same pattern applies to the exact
dispatched_count()checks at lines 797, 810-811, and 820-821. Those are less severe because a stale count fails only one expectation.♻️ Proposed change
- EXPECT_EQ(stall_capture.report_count.load(std::memory_order_acquire), 1); + ASSERT_EQ(stall_capture.report_count.load(std::memory_order_acquire), 1); EXPECT_EQ(stall_capture.group_slot, group.task_slot); - EXPECT_EQ(stall_capture.busy_target_count, 1u); + ASSERT_EQ(stall_capture.busy_target_count, 1u); EXPECT_EQ(stall_capture.busy_target_worker_ids[0], 1); - EXPECT_EQ(stall_capture.idle_queued_target_count, 1u); + ASSERT_EQ(stall_capture.idle_queued_target_count, 1u); EXPECT_EQ(stall_capture.idle_queued_target_worker_ids[0], 0); EXPECT_EQ(stall_capture.idle_queued_single_head_slots[0], single_a.task_slot);🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/ut/cpp/hierarchical/test_scheduler.cpp` around lines 786 - 796, Use ASSERT_EQ for the stall_capture.report_count check before validating the diagnostic fields, so the test returns immediately when no report arrives. Apply the same assertion-strengthening to the exact dispatched_count() checks at the indicated points, preserving their existing expected values and surrounding test logic.src/common/hierarchical/scheduler.h (2)
108-146: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueDocument the single-writer-thread invariant on
reservation_stall_episode_.
reservation_stall_deadline()readsreservation_stall_episode_undercompletion_mu_(inrun()), andupdate_reservation_stall()writes it underloop_mu_(indispatch_ready()). This is safe only because both calls execute exclusively on the scheduler thread. Add a short comment near the member to record this invariant, so a future change that invokes either method from another thread does not introduce a data race.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/common/hierarchical/scheduler.h` around lines 108 - 146, Add a short comment immediately above reservation_stall_episode_ documenting that it is accessed only by the scheduler thread, including its reads in reservation_stall_deadline() and writes in update_reservation_stall(). Do not change the locking or surrounding dispatch logic.
59-69: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winDocument the pointer-lifetime contract on
ReservationStallDiagnostic.
busy_target_worker_ids,idle_queued_target_worker_ids, andidle_queued_single_head_slotsare raw pointers. Inscheduler.cpp,update_reservation_stallbuilds this struct fromdispatch_result.busy_target_worker_ids.data()and similar calls, wheredispatch_resultis a temporary owned by the caller ofdispatch_ready(). The sink call is synchronous, so the pointers stay valid only for the duration of that call.Add a comment on the struct that states this constraint. A future sink implementation that stores these pointers past the callback invocation reads freed memory.
📝 Proposed documentation addition
struct ReservationStallDiagnostic { + // All pointer/count fields below reference memory owned by the caller + // of the sink and are valid only for the duration of the sink call. + // Do not store these pointers past the callback invocation. TaskSlot group_slot{INVALID_SLOT}; const int32_t *busy_target_worker_ids{nullptr};🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/common/hierarchical/scheduler.h` around lines 59 - 69, Add a documentation comment to the ReservationStallDiagnostic struct stating that its raw pointer fields are valid only during the synchronous ReservationStallSink callback and must not be retained afterward, since they reference caller-owned temporary storage.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/common/hierarchical/worker.cpp`:
- Around line 37-54: Update report_reservation_stall so the synchronously
invoked diagnostic sink never performs potentially blocking stderr operations.
Replace flockfile and repeated std::fprintf calls with an existing bounded
non-blocking native sink, or add explicit EPIPE and backpressure handling that
prevents SIGPIPE and unbounded blocking before this sink is registered.
In `@tests/ut/cpp/hierarchical/test_scheduler.cpp`:
- Around line 1189-1208: Move the EXPECT_EQ assertions for worker_a and worker_b
dispatch counts until after the optional notify_ready retry completes. Keep the
initial wait and retry as best-effort cleanup protection, then assert the final
counts before conditionally calling complete() so a successful retry does not
leave a prior test failure.
---
Nitpick comments:
In `@src/common/hierarchical/scheduler.h`:
- Around line 108-146: Add a short comment immediately above
reservation_stall_episode_ documenting that it is accessed only by the scheduler
thread, including its reads in reservation_stall_deadline() and writes in
update_reservation_stall(). Do not change the locking or surrounding dispatch
logic.
- Around line 59-69: Add a documentation comment to the
ReservationStallDiagnostic struct stating that its raw pointer fields are valid
only during the synchronous ReservationStallSink callback and must not be
retained afterward, since they reference caller-owned temporary storage.
In `@tests/ut/cpp/hierarchical/test_scheduler.cpp`:
- Around line 1179-1186: Add ASSERT_NE checks for manager_worker_a and
manager_worker_b immediately after their get_worker_by_id calls and before the
idle() polling loop, so null worker registrations fail the test safely before
dereferencing either pointer.
- Around line 786-796: Use ASSERT_EQ for the stall_capture.report_count check
before validating the diagnostic fields, so the test returns immediately when no
report arrives. Apply the same assertion-strengthening to the exact
dispatched_count() checks at the indicated points, preserving their existing
expected values and surrounding test logic.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: c6f392a1-58b3-4829-a64a-9d5720437d0d
📒 Files selected for processing (7)
docs/scheduler.mdsrc/common/hierarchical/scheduler.cppsrc/common/hierarchical/scheduler.hsrc/common/hierarchical/types.cppsrc/common/hierarchical/types.hsrc/common/hierarchical/worker.cpptests/ut/cpp/hierarchical/test_scheduler.cpp
3b3ebb1 to
fea8387
Compare
|
Rebased onto the reconciled #1612 head (wake model merged with #1541's run-partitioned queues) and addressed review feedback: Rebase
Review items
BehaviorUnchanged from the original design: structural discriminator (idle reserved target with queued single work), 5 s default threshold, one report per episode, no self-healing, no scheduling-policy change. Testing
|
fea8387 to
5f01440
Compare
6c9e21e to
1c9276f
Compare
A blocked NEXT_LEVEL group head reserves every target against single dispatch, so a target that goes idle while a sibling still runs sits on its own queued singles until the whole group launches. That is the designed all-or-nothing reservation and not a fault, but nothing distinguished it from a genuine stall, and it is invisible from outside the scheduler. Detect the structural shape — a blocked head with at least one reserved target that is idle and has a non-empty single FIFO — and report it once per episode after it persists for five seconds, carrying the group slot, busy target IDs, idle-but-queued target IDs, and their FIFO head slots. A head change or the condition clearing starts a new episode. This is diagnostic only: it does not classify the state as a deadlock, release the reservation, or change placement. The scheduler arms a wait_until deadline only while an episode is open and unreported, so a parked scheduler still parks. The sink is noexcept and runs on the dispatch path, so it formats into automatic storage and emits with one write(2). It allocates nothing — a throwing allocation there would call std::terminate under exactly the resource pressure worth diagnosing — takes no stdio lock a forked Worker child could inherit held, and leaves nothing running for process exit to race. A message that does not fit loses its tail and keeps its newline. reservation_stall_episode_ is confined to sched_thread_: update_reservation_ stall() writes it under loop_mu_ and reservation_stall_deadline() reads it under completion_mu_, which is race-free only because one thread does both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1c9276f to
a85d07d
Compare
Summary
NEXT_LEVELgroup reservation when an idle reserved target has queued single workBehavior
This is diagnostic only. It does not classify the condition as a deadlock, release the reservation, or otherwise change scheduling policy. Reporting is edge-triggered per group/stall episode and uses a native no-throw sink rather than a Python logger callback.
Warning sink
The sink is
noexceptand runs on the scheduler dispatch path, so it formats into automatic storage and emits with a singlewrite(2). It allocates nothing — a throwing allocation inside anoexceptsink would callstd::terminate, precisely under the resource pressure worth diagnosing — takes no stdio lock that a forkedWorkerchild could inherit held, and leaves nothing running for process exit to race. A message that does not fit the buffer loses its tail and keeps its newline.reservation_stall_episode_is confined tosched_thread_:update_reservation_stall()writes it underloop_mu_andreservation_stall_deadline()reads it undercompletion_mu_, which is race-free only because one thread does both.Dependency
#1612 is merged and this branch is rebased onto it. The scheduler's timed wait (
wait_until, armed only while a stall episode is open and unreported) is introduced here, not by #1612.Testing
ctesttargets,test_scheduler60/60pytest tests/ut -m "not requires_hardware"— 1026 passed, 7 skippedlen == strlen(buf)andlen <= cap - 1throughout)pre-commitclean (clang-format, clang-tidy, cpplint, markdownlint)